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ABSTRACT 


Self-driving vehicles are cars or trucks in which human drivers are never 
required to take control to safely operate the vehicle. They can possibly 
reform urban portability by giving maintainable, protected, and advantageous, 
clog free transportability. The issues like reliably perceiving traffic lights, 
signs, indistinct path markings can be overwhelmed by utilizing the innovative 
improvement in the fields of Deep Learning (DL). Here, Faster Region Based 
Convolution Neural Network (F-RCNN) is proposed for detection and 
recognition of Traffic Lights (TL) and signs by utilizing transfer learning. The 
input can be taken from the dataset containing various images of traffic signals 
and signs as per Indian Traffic Signals. The model achieves its target by 
distinguishing the traffic light and signs with its right class type. The proposed 
framework can likewise be upgraded for safe driving in spite of hazy path 
markings. 

KEYWORDS: Traffic Light (TL), Faster Region Based Convolution Neural Network 
(F-RCNN), Deep Learning (DL) 


INTRODUCTION 

The road traffic scenes are usually complex and it is difficult 
to extract the traffic parameters and detect the traffic 
anomaly with the existing methods, therefore deep leaning 
based approach is proposed to overcome these external 
factors. A deep neural network based model is proposed for 
reliable detection and recognition of traffic signs and lights 
using transfer learning. The method incorporates use of 
faster region based convolutional network (R-CNN) Inception 
V2 model in Tensor Flow for transfer learning. Transfer 
learning is a machine learning technique where there is 
enhancement in learning of a new task by channeling the 
knowledge through a related task that has formerly been 
learned. SSD algorithm is used for object detection with a 
deep neural network and training a high capacity model by 
using F-RCNN is able to recognize the category in which the 
detected object belongs. For the Traffic Sign detection 
considering the application requirement and available 
computational resources, Faster R-CNN Inception-V2 model 
is used which serves the accuracy and speed trade-off. 

RELATEDWORKS 

Various studies and related works can be done in the case of 
objection detection and recognition. Strategies for TL location 
are regularly delegated image preparing based, AI based or 
map-based methods [5]. In the picture handling based 
technique, a solitary or numerous measures of activities or 
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tasks are performed on the picture so as to accomplish a 
specific resultant. 

In the paper [3] AbduladhemAbdulkareem Ali, Hussein Alaa 
Hussein proposed a method which is alternative to image 
processing systems. This system is based on passive Radio 
Frequency Identification (RFID) technology. Autonomous 
driving vehicles have become a trend in the vehicle industry. 
Many driver assistance systems (DAS) have been presented 
to support these automatic cars. Traffic light Recognition 
plays an important role in these systems. This paper presents 
a RFID based system is used to recognize traffic lights status. 
The proposed approach avoids the issues that usually appear 
with the ordinary traffic lights recognition systems, especially 
with Systems that employ image processing techniques to 
recognize the traffic lights status. The approach provides a 
high performance and a low-cost system and also presents a 
RFID based system used to recognize road signs. This system 
is based on passive Radio Frequency Identification (RFID) 
technology. Using this technology, complex processing 
algorithms are not required because the signal has a small 
RFID Tag that carries all information needed for the 
inventory. A wireless transmission device suchastheX-Beeis 
adopted in this paper. Furthermore, RFID technology has few 
other advantages, they are considered as low-cost and low- 
power devices, which mean that the RFID passive tags can be 
operating for many years without battery. In order to prove 
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the systems perform properly, the system has been 
experimentally tested for both based traffic light system and 
road signs. During run tests all signals are recognized 
correctly 

In the paper [4] Shraddha Mane, Prof. Supriya Mangala 
proposed a method for moving object detection usingTensor 
Flow object detection API .In this system, the video is feed to 
the system as an input. Frames are extracted for further 
processing. The two main algorithms object detection and 
object tracking is process through deep learning methods. 
The object detection using computer vision algorithm is 
affected by different aspects like light variation, illumination, 
occlusion and system has difficulty to detect the multiple 
objects. Here, Tensor flow based object detection algorithm 
has been used. TensorFlow based object detection API is an 
open source platform. It is built on the top of TensorFlow 
which make simple to construct, train and detection models. 
In this approach, firstly the necessary libraries are imported. 
Then import the pre-trained object detection model. The 
weights are initializing along with box and tensor class. After 
initialization of all the parameters of the tensor flow model, 
the image in which object to be detected is read. Apply the 
loaded tensor flow model on the image, the TensorFlow 
based model test the image and return the location (x, y, w, h) 
of the object in the image. This is the process of object 
detection of TensorFlow object detection algorithm. The 
success rate of this approach is better and it is applicable to 
RGB images. After detecting the object, their locations are 
important to start the tracking process. After detecting the 
object, their locations are important to start the tracking 
process. Instead of using conventional computer vision based 
algorithm, in this approach Convolutional Neural Network 
(CNN) based tracking algorithm is used. 

The quality of life of people is increasing together with the 
developing technologies. One of the most important factors 
affecting daily life is smart cities. The quality of life of people 
is positively affected by emerging this concept in recent 
years. Autonomous vehicles confront with the term of the 
smart city and have become even more popular in recent 
years. In this study[4], a system of traffic lights detection and 
recognition is performed in order to reduce the accidents 
caused by traffic lights. In the paper [5] Ziya Ozcelik, Canan 
Tastimur, Mehmet Karakose, Erhan Akin proposed a method 
which has divided into two sections. Each of these parts 
requires hardware. The first part requires a camera to get the 
image. The other part requires a computer to process the 
received images. In the proposed method, images have been 
taken using CCD camera in the first step.Image processing 
techniques are performed step by step to detect the traffic 
lights in the received image through the computer. When 
traffic lights are detected, the received RGB image is 
converted into HSV format to perform chromatic separation 
from uniform and non-chromatic elements in the image. By 
performing a color based segmentation process on the 
obtained HSV format image, the locations of traffic lights in 
the image are easily detected. The color of the traffic light is 
easily determined through the SVM (Support Vector 
Machines) classification model which is a machine learning 
algorithm prepared beforehand, after the location of the 
traffic lights is determined in the image 

In the paper [6] Shaoqing Ren, Kaiming He, Ross Girshick, and 
Jian Sun presents Region Proposal Network (RPN) thatshares 


full-image convolutional features with the detection network, 
thus enabling nearly cost-free region proposals. State-of-the- 
art object detection networks depend on region proposal 
algorithms to hypothesize object locations. Advances like 
SPPnet and Fast R-CNN have reduced the running time of 
these detection networks, exposing region proposal 
computation as a bottleneck. In this work, we introduce a 
Region Proposal Network (RPN) that shares full-image 
convolutional features with the detection network, thus 
enabling nearly cost-free region proposals. An RPN is a fully 
convolutional network that simultaneously predicts object 
bounds and objectness scores at each position. The RPN is 
trained end-to-end to generate high-quality region proposals, 
which are used by Fast R-CNN for detection. We further 
merge RPN and Fast R-CNN into a single network by sharing 
their convolutional features—using the recently popular 
terminology of neural networks with "attention” 
mechanisms, the RPN component tells the unified network 
where to look. 

Dwi H. Widyantoro and Kevin I. Saputra accomplished TL 
discovery and acknowledgment with the assistance of Color 
Segmentation and Circle Hough Transform [8] while Guo Mu 
accomplished the equivalent with RGB to HSV change, sifting, 
histogram of situated slopes (HOG) highlights and bolster 
vector machine (SVM) [9]. Swathy S Pillai built a framework 
identifying tail lights for breaking down traffic during late 
evening utilizing different morphological tasks like 
thresholding, separating, extraction, and so on [21]. Zhenwei 
Shi talked about Adaptive Background Suppression channels 
as a quick and hearty technique for TL discovery under 
various brightening conditions [22].In spite of the fact that 
picture handling approach is very straight and 
straightforward, it experiences basic stages, for example, 
thresholding, separating. Fine miscounts or slight deviations 
from measures in these stages may prompt equivocal results 
which is carefully unfortunate in delicate instances of TL 
location. To handle this entanglement, AI based strategies 
and calculations are attempted over separately or in blend 
with abundant preparing procedures to prune the deceptive 
headings. For instance, Keyu Lu proposed Generalized Haar 
Filter based Convolution Neural Network (CNN) plausible to 
be conveyed for object location in rush hour gridlock scenes 
[15]. Seokwoo Jung created CNN-based traffic sign 
acknowledgment calculation where extraction of traffic sign 
up-and-comers is acted in first stage and grouping with 
LeNet-5 CNN engineering happens in further stage [16]. 
Gwang-Gook. LEE and Byung Kwan PARK accomplished 
steadfast results for TL acknowledgment by consolidating 
ordinary methodology of picture handling with Deep Neural 
Network (DNN) as a promising classifier [17]. Likewise, 
Karsten Behrendt set forth profound learning approach 
consolidating sound system vision and vehicle odometry for 
TL discovery, following and arrangement [18]. Masamitsu 
Tsuchiya uncovered an effective Hybrid Transfer Learning 
strategy for object identification offering helps to release the 
untried learning techniques [19]. 

In the paper [9] Sang-HyukLee, Jung-Hawn Kim, Yong-JinLim 
and Joonhong Lim proposed an algorithm for traffic light 
detection and recognition based on Haar-like features. Haar- 
like features are used to learn about the traffic light image 
and detect the candidate area based on the learning data. The 
detected candidate image is verified by the pre-learned SVM 
(Support Vector Machine) classifier, and binarization and 
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morphology operations are performed on the verified 
candidate image for detection of the traffic light object. The 
detected traffic light is divided into respective signal areas to 
determine the current on/off status of traffic lights. The 
signal signs in the respective areas are defined by regulation 
and the sign of traffic lights can be recognized by recognizing 
on/off of the signals in the respective areas. 

In the paper [10] Priyanka D, Dharani K, Anirudh C, Akshay K 
presents a Traffic Sign Recognition system to regulate traffic 
signs, warn a driver and command certain actions. Fast 
robust and real-time automatic traffic sign detection and 
recognition can support the driver and significantly increase 
driving safety. Automatic recognition of traffic signs is also 
important for an automated intelligent driving vehicle or for a 
driver assistance system. This is a visual based project i.e., the 
input to the system is video data which is continuously 
captured from the webcam is interfaced to the Raspberry 
Pi.Images are pre-processed with several image processing 
techniques such as; Hue, Saturation and Value (HSV) color 
space model technique is employed for traffic light detection, 
for sign detection again HSV color space model and Contour 
Algorithm has been used. The signs are detected based on 
Region of Interest (ROI).The ROI is detected based on the 
features like geometric shape and color of the object in the 
image containing the traffic signs. The experimental results 
show highly accurate classifications of traffic sign patterns 
with complex background images as well as the results 
accomplish in reducing the computational cost of this 
proposed method. 

In the paper [11] Chunhe Yu, Chuan Huang, Yao Lang 
presents a method using image processing technology .In 
order to reduce accident at traffic intersections during day 
and night, the algorithm of traffic lights detection which is 
applied in a vehicle driver assistance system is designed by 
using the image processing technology. The system of traffic 
light detection includes three parts: a CCD camera, an image 
acquisition card, and a PC. Based on RGB color space, the 
algorithm extracts red, green, and yellow objects in the image 
firstly; For the purpose of eliminating disturbance in the 
environment, the features of traffic lights are used to verify 
the object identity, and then the types of traffic signals are 
judged The results of experiments show that the algorithm is 
stable and reliable. The work process is as follows: As a 
vehicle travels at non-intersection [no traffic lights in the 
scene), the program of road curb detection operates, while it 
travels at intersection (traffic lights in the scene), the 
program of pedestrian detection operates, and if there is a 
hazardous event occurring, the assistance driver system give 
a warning signal to the driver. 

In the paper [12] presents a software technique based on HSV 
color model. Automatic traffic light violation detection system 
relies on color detection of traffic light appeared in video 
frames. Presence of red light in video frame triggers detection 
software routine to identify vehicles violating traffic light. 
Detection of red light in video frames can be difficult due to: 
fading or dimming of red light, obscurity from large vehicles 
and flare. Here ,a software technique based on HSV (Hue, 
Saturation, Value) color model is proposed to eliminate 
difficulties in red light detection mentioned above and is able 
to identify all colors of traffic light which gives 96% detection 
accuracy. The proposed scheme includes 3 Methods: First 
method Converts video frames to grayscale image, and then 


followed by thresholding which yields binary images. All 
three colors of traffic light are compared with predefined 
pattern to determine which one is on. If there are more than 
one color of traffic light found to be on, the threshold value is 
adjusted and the process is repeated. Method 2: Convert 
video frames from RGB to HSV color system, then compare 
value of hue to determine color of traffic light. Method 3: 
Combine result of above methods. The color of traffic light is 
produced when the results from two methods are identical. 

In the paper [13] Zhenwei Shi, Member, IEEE, Zhengxia Zou, 
and Changshui Zhang proposed A novel vision-based traffic 
light detection method for driving vehicles, which is fast and 
robust under different illumination conditions. The proposed 
method contains two stages: the candidate extraction stage 
and the recognition stage. On the candidate extraction stage, 
we propose an adaptive background suppression algorithm 
to highlight the traffic light candidate regions while 
suppressing the undesired backgrounds. On the recognition 
stage, each candidate region is verified and is further 
classified into different traffic light semantic classes. We 
evaluate our method on video sequences (more than 5000 
frames and labels) captured from urban streets and suburb 
roads in varying illumination and compared with other vision 
based traffic detection approaches. 

In the paper [14] Ying Li, Lingfei Ma, Yuchun Huang, Jonathon 
Li presents a segment-based traffic sign detection method by 
using vehicle-borne mobile laser scanning (MLS) data. This 
method has three steps: road scene segmentation, clustering 
and traffic sign detection. The non-ground points are firstly 
segmented from raw MLS data by estimating road ranges 
based on vehicle trajectory and geometric features of roads 
(e.g., surface normals and planarity). The ground points are 
then removed followed by obtaining non-ground points 
where traffic signs are contained. Secondly, clustering is 
conducted to detect the traffic sign segments (or candidates) 
from the non-ground points. Finally, these segments are 
classified to specified classes. Shape, elevation, intensity, 2D 
and 3D geometric and structural features of traffic sign 
patches are learned by the support vector machine (SVM) 
algorithm to detect traffic signs among segments. The 
proposed algorithm has been tested on a MLS point cloud 
dataset acquired by a Leador system in the urban 
environment. The results demonstrate the applicability of the 
proposed algorithm for detecting traffic signs in MLS point 
clouds. 

METHEDOLOGY 

The proposed system is mainly used for detection and 
recognition of Traffic Light with the help of object detection 
method and faster region based convolution neural network 
(F-RCNN).Convolution Neural Network (CNN) is one of the 
most popular ways of doing object recognition. It is widely 
used and most state-of-the-art neural networks used this 
method for various object recognition related tasks such as 
image classification. This F-RCNN network takes an image as 
input and outputs the probability of the different classes. If 
the object present in the image then its output probability is 
high else the output probability of the rest of classes is either 
negligible or low. 

The input must be an image and the goalis: 

1. Dataset Preparation 

2. Loading to Tensor Flow 
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3. Input custom image 

4. Object Detection 

5. Traffic Light Detection in Tensor Flow 

6. Display Output 

There are Four modules in this project, Data Preparation 
Module, Making Inception Model, Object Detection, Traffic 
Light Recognition. 

A. Data Preparation Module 

The first part of the proposed system is data preparation 
module. For deep learning natural language project first of all 
we can convert the input documents as integers. The first 
part of the proposed system is data preparation module. The 
deep learning system, in both its training and operational 
modes, begins with the input of a Traffic Light image that 
defines the object to be classified. The Traffic Light images 
with almost same resolution are used as the input for the 
easiness of proper training and testing. 

B. Making Inception Model 

Transfer learning is a machine learning method which utilizes 
a pre-trained neural network. For example, the image 
recognition model called Inception-v3 model, is a 
convolutional neural network that is trained on more than a 
million images from the Image Net database. The network is 
48 layers deep and can classify images into 1000 object 
categories, such as keyboard, mouse, pencil, and many 
animals and here it will be classified into RED, GREEN, 
YELLOW, GO SLOW, SPEED LIMIT 80, NO PARKING, ZEBRA 
CROSSING etc... In transfer learning, when we build a new 
model to classify our original dataset, we reuse the feature 
extraction part and re-train the classification part with our 
dataset. 

C. Object Detection 

Single Shot MultiBox Detector (SSD) is designed for object 
detection in real time. Faster R-CNN uses a region proposal 
network to create boundary boxes and utilizes those boxes to 
classify objects. While it is considered the start-of-the-art in 
accuracy, the whole process runs at 7 frames per second. SSD 
speeds up the process by eliminating the need of the region 
proposal network. To recover the drop in accuracy, SSD 
applies a few improvements including multi-scale features 
and default boxes. These improvements allow SSD to match 
the Faster R-CNN's accuracy using lower resolution images, 
which further pushes the speed higher, it achieves the real 
time processing speed 

D. Traffic Light Detection 

Faster RCNN is an object detection architecture that uses 
convolution neural networks like YOLO and SSD. Faster 
RCNN is formed from 3 parts: In convolution layers we train 
filters to extract the appropriate features from the image, for 
example we are going to train those filters to extract the 
appropriate features for a traffic light, sign and then those 
filters are going to learn through training shapes and colors 
that only exist in the traffic light related image. Pooling 
consists of decreasing quantity of features in the features 
map by eliminating pixels with low values.RPN is small 
neural network sliding on the last feature map of the 
convolution layers and predict whether there is an object or 
not and also predict the bounding box of those objects. Now 
we use other fully connected neural networks that takes as an 
input the regions proposed by the RPNand predict object 
class (classification) and Bounding boxes (Regression). 


EXPERIMENTAL RESULTS 
A. Results 

This section discusses the experimental results of the 
proposed system. The proposed system is using LIZA traffic 
light dataset which are taken from kaggle for results 
assessment. Here the dataset taken contains 25 categories of 
images with traffic light and signs. For each category it should 
contain more than 40 images otherwise it may cause issues. 
First we upload an image. 



Figure 1. Selecting Image to upload 

Image uploaded successfully, and the uploaded image first 
checks out if it is traffic light, sign or not and the detection is 
carried out by using SSD After detection of Traffic Light, itwill 
pass to the Inception Model in order to find in which category 
the detected object belongs. 



Traffic Signal Prediction 
using Deep Learning 


Upload image of 
your Image 


Browse... j 050_0014.jpg 


PREDICT 


Figure 2. Image Uploaded Successfully 



Figure 3: Predicted Image Category 
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For those objects which have the maximum probability value 
is taken as its final class category. Here the model 
successfully detects the traffic sign "RAILWAY CROSSING" 
with accuracy 83.16%. Experimental results show that the 
proposed SSD method produces an overall accuracy of 
96.6%. 

PERFORMANCEREQUIREMENTS 

Performance requirements describe all the hardware 
specification of the system. For deep learning project, the 
minimum system requirement is, it needs i3 or above 
processor. Mostly deep learning projects used 64 bit Ubuntu 
operating system. Here we used a Mac OS with 51.8 GHz Intel 
Core i5 processor for this project (For Mac OS only 10.12.6 
Sierra or later os versions must be used). Our Mac OS 
contains 4GB RAM, 160GB or above hard drive space. 

CONCLUSION 

A driverless car is a vehicle capable of sensing its 
environment and operating without human involvement. An 
autonomous car can go anywhere a traditional cargoes and 
do everything that an experienced human driver does. In the 
field of self-driving vehicles, the proposed work fills in as a 
module for navigation system. For driverless cars, Traffic 
light object detection is to be out carried out and they can be 
implemented by using deep learning approach. The use of 
Faster R-CNN Inception-V2 model via transfer learning 
improves the accuracy which makes the system reliable for 
real time application. The outcomes with bounding boxes 
provide guidelines for real time control actions of the vehicle. 
In this project only a dataset with 25 categories is selected for 
processing. The dataset created for the system covers various 
images of traffic light according to the Indian Traffic System. 
In advancement to this, the system can also be optimized for 
safe driving despite unclear lane markings. Also, it can be 
equipped with the ability to respond to spoken commands or 
hand signals from law enforcement or highway safety 
employees. 
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